A Visually Enhanced Neural Encoder for Synset Induction

The synset induction task is to automatically cluster semantically identical instances, which are often represented by texts and images. Previous works mainly consider textual parts, while ignoring the visual counterparts. However, how effectively employ information enhance semantic representation for challenging. In this paper, we propose a Visually Enhanced NeUral Encoder (i.e., VENUE) learn multimodal task. key insight lies in construct representations through intra-modal inter-modal interactions among images text. Specifically, first design interaction module attention mechanism capture correlation To obtain multi-granularity representations, fuse pre-trained tags word embeddings. Second, masking filter out weakly relevant information. Third, present gating adaptively regulate modalities’ contributions semantics. A triplet loss adopted train VENUE encoder learning discriminative representations. Then, perform clustering algorithms on obtained induce synsets. verify our approach, collect dataset, i.e., MMAI-Synset, conduct extensive experiments. experimental results demonstrate that method outperforms strong baselines three groups of evaluation metrics.